Estimation Techniques

Estimates can be made at a number of levels, as shown in the figure below.

Estimates for a MTP are created early on in the project. Often, not all knowledge of the test object is available at this point. As a consequence, the accuracy of the estimate is limited. The size and complexity of the test object may change during the project. It is important for the test manager to make it clear to the stakeholders that the estimate is based on a number of assumptions and therefore details will have to be added later. A possible solution is to use margins to represent the initial estimate for an MTP.

The estimate in the MTP constitutes the framework for the estimates per test level (e.g. system test, user acceptance test, and production acceptance test). The required time for the various phases – Control, Setting up and maintaining infrastructure, Preparation, Specification, Execution and Completion – is then established for the test level. Separate test activities are estimated within the test phases. The time necessary to create the MTP (Planning) is not included in the estimates. A fixed number of hours is usually estimated for this. After all, establishing the plan consists of executing clearly defined activities. The impact of e.g. the test object size on the time required to create the MTP is limited in this context. If there is an impact, it will be noticeable. In practice, some 60 to 160 hours are usually invested in creating the MTP.

As the estimate is made later in the test process and therefore at a lower level, more knowledge of the test object is available. Moreover, experiences from earlier on in the process can be used, making the estimate more accurate.

Independent of the level, creating the plan consists of the following generic steps:

Inventory the available material that can serve as a basis for the estimate.
Select (a number of) estimating techniques - We recommend using multiple techniques in parallel. This makes it possible to compare the outcome of the various techniques. In addition to estimating techniques, it is worthwhile asking an experienced employee to make an estimate of the required time (expert estimate).
Determine the definitive estimate - The aim of this step is to combine the outcomes of the previous step into one single estimate. If the outcomes vary little, taking an average will work. In other cases the differences have to be analysed. If an adequate estimate cannot be made after analysing the differences, the client must be consulted. The test manager explains the problems and makes proposals to achieve a correct estimate.
Present the outcome - The aim of presenting the outcome is to provide insight to the business into the consequences of the selected test strategy and approach. It is important to show clearly which assumptions were made. Especially with an estimate created very early on in the process, assumptions will be involved that will become more concrete later on in the process.

As discussed earlier, there are various estimating techniques to create an estimate. Choosing the right ones in particular is a step requiring experience. The sections below describe the estimating techniques, based on the following assumptions:

Estimating the test activities in the development phase (unit test and unit integration test) is an integrated component in estimating the realisation project and is not taken into consideration unless explicitly specified.
Where possible, experience figures are mentioned for the specified techniques. We explain the background of these figures. The figures shown must always be considered within the described context. They do not necessarily apply in a different situation.
One retest is included in all of the experience figures mentioned in subsequent sections.

An adequate choice from the various techniques can be made with the use of two tables. These tables answer the following two questions:

Which technique is suitable for which level of estimating?
Which techniques are suitable for estimating which quality characteristics?

The answers to these questions are shown in the tables below.

	Master test plan	Detailed test plan	Test phase	Test activity
Estimating based on ratios	X	X	X	(X)
Estimating based on size	X
Work Breakdown Structure (WBS)	X	X	X	X
Evaluation estimating techniques	X
Proportionate estimation	X	X	X	(X)
Extrapolation			X	X
Test point analysis (estimating based on size and strategy)	X	X

The possible estimating techniques are shown per quality characteristic in the table below. The table distinguishes between three different levels of testing depth for dynamic tests, i.e.

and

(low, medium and high).

	Evaluation	Statistical Test	UT	UIT	Implicit Test	Dynamic Test	Dynamic Test	Dynamic Test
Depth of testing →	●	●	●	●		●	●●	●●●
Quality characteristic↓	No. of pages 1)		2)	2)	3)
Connectivity		TPA-s				-	-	-
Continuity		TPA-s				Timebox7) week	Timebox7) month	Timebox7) quarter
Data controllability		TPA-s				-	-	-
Effectivity		TPA-s				TPA6)	TPA6)	TPA6)
Efficiency		TPA-s				-	WBS	-
Flexibility		TPA-s				-	-	-
Functionality		TPA-s	Hour box7)	Hour box7)		TPA	TPA	TPA
Infrastructure		TPA-s				-	-	-
Manageability		TPA-s 4)				WBS5)	-	-
Maintainability		TPA-s				-	-	-
Performance Batch Online		TPA-s				TPA WBS	TPA WBS	TPA WBS
Portability		TPA-s				WBS	TPA	TPA
Reusability		TPA-s				-	-	-
Security		TPA-s				TPA	TPA	WBS
Suitability		TPA-s				TPA6)	TPA6)	TPA
Testability		TPA-s				-	-	-
User-friendliness		TPA-s				WBS	WBS	WBS

Comments on the table:
It is not possible to indicate a specific estimating technique for this level of depth.

Several pages must be read when verifying a quality characteristic. Quality characteristics that have to do with functionality require a study of the pages on which the functionality is described. Other quality characteristics are generally described on other pages. This results in a varying number of pages per quality characteristic for verification.
It is assumed that the estimate of the standard test activities in the UT and UIT is part of the estimate of the realisation. If desirable, extra attention to testing during the UT and UIT can be specified. The estimating technique for this is an hour box, in which e.g. a supplement rate is added to the build effort (e.g. 10%) or part of the effort for the ST.
TPA-i is the component for implicit testing of a quality characteristic during dynamic testing of another quality characteristic. In TPA, this results in an additional supplement of 0.02 when determining the Qd.
TPA-s is the statistical component of TPA.
WBS = Work Breakdown Structure.
If effectivity and suitability are tested with the same test type/test technique, the effort is included once.
The time box and hour box are determined by factors outside the test process. Time box week in the table above means that testing takes a period of one week